Displaying handwritten strokes on a device according to a determined stroke direction matching the present direction of inclination of the device

ABSTRACT

According to one embodiment, an electronic device includes a display processor, a determiner and a storage processor. The display processor displays on a screen a plurality of strokes input by handwriting. The determiner determines a top-and-bottom direction for each of the plurality of strokes. The storage processor is configured to store in a storage medium handwritten data. The handwritten data includes a plurality of stroke data corresponding to the plurality of strokes, and a plurality of pieces of direction information relating to a top-and-bottom direction each of the plurality of strokes.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2013-091500, filed Apr. 24, 2013, theentire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to a technique ofprocessing handwritten data.

BACKGROUND

In recent years, various kinds of electronic devices, such as a tabletterminal, a PDA and a smartphone, have been developed. Most of theseelectronic devices include touch-screen displays for facilitating inputoperations by users.

By touching a menu or an object, which is displayed on the touch-screendisplay, by a finger or the like, the user can instruct an electronicdevice to execute a function which is associated with the menu orobject.

However, most of existing electronic devices with touch-screen displaysare consumer products which are designed to enhance operability onvarious media data such as video and music, and are not necessarilysuitable for use in a business situation such as a meeting, a businessnegotiation or product development. Thus, in business situations,paper-based pocket notebooks have still been widely used.

Recently, a technique for determining a relationship between a tabletterminal and the seating position of a user, based on the direction ofwriting of handwritten characters, has also been developed.

In the meantime, in general, most of tablet terminals have a function ofautomatically rotating the direction of a screen image in accordancewith the direction of the tablet terminal.

However, in a handwriting application which handles a handwrittendocument, if the direction of the screen image of the application isautomatically rotated in accordance with the direction of the tabletterminal, there may be a case in which a feeling of use, such as afeeling of using a real paper-based pocket notebook, cannot be obtained.The reason for this is that if the direction of the screen image isautomatically rotated, it would become difficult to handwritecharacters, etc. in a free direction on the display screen of thehandwriting application.

On the other hand, the handwriting application is required to also havea function as a digital tool for handling a handwritten document asdigital data.

BRIEF DESCRIPTION OF THE DRAWINGS

A general architecture that implements the various features of theembodiments will now be described with reference to the drawings. Thedrawings and the associated descriptions are provided to illustrate theembodiments and not to limit the scope of the invention.

FIG. 1 is an exemplary perspective view illustrating an externalappearance of an electronic device according to an embodiment.

FIG. 2 is an exemplary view illustrating a cooperative operation betweenthe electronic device of the embodiment and an external apparatus.

FIG. 3 is a view illustrating an example of a handwritten document whichis handwritten on a touch-screen display of the electronic device of theembodiment.

FIG. 4 is an exemplary view for explaining time-series informationcorresponding to the handwritten document of FIG. 3, the time-seriesinformation being generated by the electronic device of the embodiment.

FIG. 5 is an exemplary view for describing a note view screen which isdisplayed by the electronic device of the embodiment.

FIG. 6 is an exemplary view for describing an operation of displayingstrokes which are handwritten in a state in which the electronic deviceof the embodiment is inclined such that the direction of the note viewscreen agrees with the direction of gravity.

FIG. 7 is an exemplary view for describing an operation of displayingstrokes which are handwritten in a state in which the electronic deviceof the embodiment is inclined such that the direction of the note viewscreen is opposite to the direction of gravity.

FIG. 8 is an exemplary view for explaining handwritten data includingstroke data and stroke direction information stored by the electronicdevice of the embodiment.

FIG. 9 is an exemplary view for describing an operation of displayingstrokes which are handwritten in a state in which the electronic deviceof the embodiment is inclined such that the direction of the note viewscreen is inclined by 90° to the left, relative to the direction ofgravity.

FIG. 10 is an exemplary view for describing an operation of displayingstrokes which are handwritten in a state in which the electronic deviceof the embodiment is inclined such that the direction of the note viewscreen is inclined by 90° to the right, relative to the direction ofgravity.

FIG. 11 is an exemplary view for explaining stroke direction informationcorresponding to a stroke series having a direction agreeing with thedirection of the note view screen.

FIG. 12 is an exemplary view for explaining stroke direction informationcorresponding to a stroke series having a direction opposite to thedirection of the note view screen.

FIG. 13 is an exemplary view for explaining an example of directions ofthree axes of an acceleration sensor in the electronic device of theembodiment.

FIG. 14 is an exemplary view for explaining a relationship between theinclination direction of the electronic device of the embodiment and adetection output of the acceleration sensor.

FIG. 15 is a view illustrating an example of an operation of determiningstroke direction information with use of a face recognition result.

FIG. 16 is a view illustrating another example of the operation ofdetermining stroke direction information with use of a face recognitionresult.

FIG. 17 is a view illustrating still another example of the operation ofdetermining stroke direction information with use of a face recognitionresult.

FIG. 18 is a view illustrating still another example of the operation ofdetermining stroke direction information with use of a face recognitionresult.

FIG. 19 is an exemplary block diagram illustrating a systemconfiguration of the electronic device of the embodiment.

FIG. 20 is an exemplary block diagram illustrating a functionalconfiguration of a digital notebook application program which isexecuted by the electronic device of the embodiment.

FIG. 21 is an exemplary block diagram for explaining a characterrecognition operation on strokes having a direction corresponding to anormal direction, the character recognition operation being executed bythe electronic device of the embodiment.

FIG. 22 is an exemplary block diagram for explaining a characterrecognition operation on strokes having a direction opposite to thenormal direction, the character recognition operation being executed bythe electronic device of the embodiment.

FIG. 23 is an exemplary view for explaining the outline of a handwritingsearch which is executed by the electronic device of the embodiment.

FIG. 24 is an exemplary view for explaining a handwriting searchoperation for strokes having a direction corresponding to the normaldirection, the handwriting search operation being executed by theelectronic device of the embodiment.

FIG. 25 is an exemplary view for explaining a handwriting searchoperation for strokes having a direction opposite to the normaldirection, the handwriting search operation being executed by theelectronic device of the embodiment.

FIG. 26 is a view for describing an example of a handwritten documentincluding both strokes having a direction corresponding to the normaldirection and strokes having a direction opposite to the normaldirection, the handwritten document being displayed by the electronicdevice of the embodiment.

FIG. 27 is a view for describing an example of an operation ofselectively displaying either strokes having a direction correspondingto the normal direction or strokes having a direction opposite to thenormal direction, in accordance with the present inclination directionof the electronic device of the embodiment.

FIG. 28 is a view for describing another example of the operation ofselectively displaying either strokes having a direction correspondingto the normal direction or strokes having a direction opposite to thenormal direction, in accordance with the present inclination directionof the electronic device of the embodiment.

FIG. 29 is an exemplary flowchart illustrating the procedure of ahandwritten data storage process which is executed by the electronicdevice of the embodiment.

FIG. 30 is an exemplary flowchart illustrating the procedure of ahandwritten data display process which is executed by the electronicdevice of the embodiment.

DETAILED DESCRIPTION

Various embodiments will be described hereinafter with reference to theaccompanying drawings.

In general, according to one embodiment, an electronic device includes adisplay processor, a determiner and a storage processor. The displayprocessor is configured to display on a screen a plurality of strokesinput by handwriting. The determiner is configured to determine atop-and-bottom direction for each of the plurality of strokes. Thestorage processor is configured to store in a storage medium handwrittendata. The handwritten data includes a plurality of stroke datacorresponding to the plurality of strokes, and a plurality of pieces ofdirection information relating to a top-and-bottom direction of each ofthe plurality of strokes. The determiner is configured to determine atop-and-bottom direction of each of the plurality of strokes, by usingat least either a direction of inclination of the electronic devicewhich is determined in accordance with an output of an accelerationsensor in the electronic device, or a positional relationship betweenthe electronic device and a user, which is determined in accordance witha result of face recognition using an image captured by a camera moduleof the electronic device.

FIG. 1 is a perspective view illustrating an external appearance of anelectronic device according to an embodiment. The electronic device is,for instance, a pen-based portable electronic device which can execute ahandwriting input by a pen or a finger. This electronic device may berealized as a tablet computer, a notebook-type personal computer, asmartphone, a PDA, etc. In the description below, the case is assumedthat this electronic device is realized as a tablet computer 10. Thetablet computer 10 is a portable electronic device which is also called“tablet” or “slate computer”. As shown in FIG. 1, the tablet computer 10includes a main body 11 and a touch-screen display 17. The main body 11has a thin box-shaped housing. The touch-screen display 17 is attachedsuch that the touch-screen display 17 is laid over the top surface ofthe main body 11. In addition, a camera module (Web camera) 18 isdisposed on the top surface of the main body 11. The camera module 18 isprovided, for example, at an upper end portion of the main body 11,which is located on the upper side of the touch-screen display 17. Thecamera module 18 can capture an image (a moving picture or a stillimage) in the vicinity of the front side of the tablet computer 10.Using the camera 18, a user can capture an image (a moving picture or astill image) or can take a photo.

In the touch-screen display 17, a flat-panel display and a sensor, whichis configured to detect a touch position of a pen or a finger on thescreen of the flat-panel display, are assembled. The flat-panel displaymay be, for instance, a liquid crystal display (LCD). As the sensor, forexample, use may be made of an electrostatic capacitance-type touchpanel, or an electromagnetic induction-type digitizer. In thedescription below, the case is assumed that two kinds of sensors, namelya digitizer and a touch panel, are both assembled in the touch-screendisplay 17.

The digitizer is disposed, for example, under the screen of theflat-panel display. The touch panel is disposed, for example, over thescreen of the flat-panel display. The touch-screen display 17 can detectnot only a touch operation on the screen with use of a finger, but alsoa touch operation on the screen with use of a pen 100. The pen 100 maybe, for instance, an electromagnetic-induction pen. The user can executea handwriting input operation on the touch-screen display 17 by using anexternal object (pen 100 or finger). During the handwriting inputoperation, a locus of movement of the external object (pen 100 orfinger) on the screen, that is, a locus (trace of writing) of a strokewhich is handwritten by a handwriting input operation, is drawn in realtime, and thereby the loci of respective strokes are displayed on thescreen. A locus of movement of the external object during a time inwhich the external object is in contact with the screen corresponds toone stroke. A set of many strokes corresponding to handwrittencharacters or handwritten graphics, that is, a set of many loci (tracesof writing), constitutes a handwritten document.

In the present embodiment, this handwritten document is stored in astorage medium not as image data but as time-series informationindicative of coordinate series of the loci of strokes and the orderrelation between the strokes. The details of this time-seriesinformation will be described later with reference to FIG. 4. Thistime-series information indicates an order in which a plurality ofstrokes are handwritten, and includes a plurality of stroke datacorresponding to a plurality of strokes. In other words, the time-seriesinformation means a set of time-series stroke data corresponding to aplurality of strokes. Each stroke data corresponds to one stroke, andincludes coordinate data series (time-series coordinates) correspondingto points on the locus of this stroke. The order of arrangement of thesestroke data corresponds to an order in which strokes are handwritten,that is, an order of strokes.

The tablet computer 10 can read out arbitrary existing time-seriesinformation (handwritten data) from the storage medium, and can displayon the screen a handwritten document corresponding to this time-seriesinformation, that is, a plurality of strokes indicated by thistime-series information. Furthermore, the tablet computer 10 has an editfunction. The edit function can delete or move an arbitrary stroke or anarbitrary handwritten character or the like in the displayed handwrittendocument, in accordance with an edit operation by the user with use ofan “eraser” tool, a range designation tool, and other various tools.Besides, this edit function includes an operation of clearing thehistory of some handwriting operations.

In this embodiment, the time-series information (handwritten document)may be managed as one page or plural pages. In this case, thetime-series information (handwritten document) may be divided in unitsof an area which falls within one screen, and thereby a piece oftime-series information, which falls within one screen, may be stored asone page. Alternatively, the size of one page may be made variable. Inthis case, since the size of a page can be increased to an area which islarger than the size of one screen, a handwritten document of an arealarger than the size of the screen can be handled as one page. When onewhole page cannot be displayed on the display at a time, this page maybe reduced in size and displayed, or a display target part in the pagemay be moved by vertical and horizontal scroll.

FIG. 2 shows an example of a cooperative operation between the tabletcomputer 10 and an external apparatus. The tablet computer 10 cancooperate with a personal computer 1 or a cloud. Specifically, thetablet computer 10 includes a wireless communication device of, e.g.wireless LAN, and can wirelessly communicate with the personal computer1. Further, the tablet computer 10 can communicate with a server 2 onthe Internet. The server 2 may be a server which executes an onlinestorage service, and other various cloud computing services.

The personal computer 1 includes a storage device such as a hard diskdrive (HDD). The tablet computer 10 can transmit time-series information(handwritten data) to the personal computer 1 over a network, and canstore the time-series information (handwritten data) in the HDD of thepersonal computer 1 (“upload”). In order to ensure a securecommunication between the tablet computer 10 and personal computer 1,the personal computer 1 may authenticate the tablet computer 10 at atime of starting the communication. In this case, a dialog for promptingthe user to input an ID or a password may be displayed on the screen ofthe tablet computer 10, or the ID of the tablet computer 10, forexample, may be automatically transmitted from the tablet computer 10 tothe personal computer 1.

Thereby, even when the capacity of the storage in the tablet computer 10is small, the tablet computer 10 can handle many pieces of time-seriesinformation (handwritten data) or large-volume time-series information(handwritten data).

In addition, the tablet computer 10 can read out (“download”) one ormore arbitrary time-series information stored in the HDD of the personalcomputer 1, and can display the locus of each stroke indicated by theread-out time-series information on the screen of the display 17 of thetablet computer 10. In this case, the tablet computer 10 may display onthe screen of the display 17 a list of thumbnails which are obtained byreducing in size the pages of plural pieces of time-series information(handwritten data), or may display one page, which is selected fromthese thumbnails, on the screen of the display 17 in the normal size.

Furthermore, the destination of communication of the tablet computer 10may be not the personal computer 1, but the server 2 on the cloud whichprovides storage services, etc., as described above. The tablet computer10 can transmit time-series information (handwritten data) to the server2 over the network, and can store the time-series information(handwritten data) in a storage device 2A of the server 2 (“upload”).Besides, the tablet computer 10 can read out arbitrary time-seriesinformation which is stored in the storage device 2A of the server 2(“download”) and can display the locus of each stroke indicated by thetime-series information on the screen of the display 17 of the tabletcomputer 10.

As has been described above, in the present embodiment, the storagemedium in which the time-series information is stored may be the storagedevice in the tablet computer 10, the storage device in the personalcomputer 1, or the storage device in the server 2.

Next, referring to FIG. 3 and FIG. 4, a description is given of arelationship between strokes (characters, marks, graphics, tables,etc.), which are handwritten by the user, and time-series information.FIG. 3 shows an example of a handwritten document (handwritten characterstring) which is handwritten on the touch-screen display 17 by using thepen 100 or the like.

In many cases, on a handwritten document, other characters or graphicsare handwritten over already handwritten characters or graphics. In FIG.3, the case is assumed that a handwritten character string “ABC” washandwritten in the order of “A”, “B” and “C”, and thereafter ahandwritten arrow was handwritten near the handwritten character “A”.

The handwritten character “A” is expressed by two strokes (a locus of“^” shape, a locus of “-” shape) which are handwritten by using the pen100 or the like, that is, by two loci. The locus of the pen 100 of thefirst handwritten “^” shape is sampled in real time, for example, atregular time intervals, and thereby time-series coordinates SD11, SD12,. . . , SD1 n of the stroke of the “^” shape are obtained. Similarly,the locus of the pen 100 of the next handwritten “-” shape is sampled inreal time, for example, at regular time intervals, and therebytime-series coordinates SD21, SD22, . . . , SD2 n of the stroke of the“-” shape are obtained.

The handwritten character “B” is expressed by two strokes which arehandwritten by using the pen 100 or the like, that is, by two loci. Thehandwritten character “C” is expressed by one stroke which ishandwritten by using the pen 100 or the like, that is, by one locus. Thehandwritten “arrow” is expressed by two strokes which are handwritten byusing the pen 100 or the like, that is, by two loci.

FIG. 4 illustrates time-series information 200 corresponding to thehandwritten document of FIG. 3. The time-series information 200 includesa plurality of stroke data SD1, SD2, . . . , SD7. In the time-seriesinformation 200, the stroke data SD1, SD2, . . . , SD7 are arranged intime series in the order of strokes, that is, in the order in whichplural strokes are handwritten.

In the time-series information 200, the first two stroke data SD1 andSD2 are indicative of two strokes of the handwritten character “A”. Thethird and fourth stroke data SD3 and SD4 are indicative of two strokeswhich constitute the handwritten character “B”. The fifth stroke dataSD5 is indicative of one stroke which constitutes the handwrittencharacter “C”. The sixth and seventh stroke data SD6 and SD7 areindicative of two strokes which constitute the handwritten “arrow”.

Each stroke data includes coordinate data series (time-seriescoordinates) corresponding to one stroke, that is, a plurality ofcoordinates corresponding to a plurality of points on the locus of onestroke. In each stroke data, the plural coordinates are arranged in timeseries in the order in which the stroke is written. For example, asregards handwritten character “A”, the stroke data SD1 includescoordinate data series (time-series coordinates) corresponding to thepoints on the locus of the stroke of the “^” shape of the handwrittencharacter “A”, that is, an n-number of coordinate data SD11, SD12, . . ., SD1 n. The stroke data SD2 includes coordinate data seriescorresponding to the points on the locus of the stroke of the “-” shapeof the handwritten character “A”, that is, an n-number of coordinatedata SD21, SD22, . . . , SD2 n. Incidentally, the number of coordinatedata may differ between respective stroke data.

Each coordinate data is indicative of an X coordinate and a Ycoordinate, which correspond to one point in the associated locus. Forexample, the coordinate data SD11 is indicative of an X coordinate (X11)and a Y coordinate (Y11) of the starting point of the stroke of the “^”shape. The coordinate data SD1 n is indicative of an X coordinate (X1n)and a Y coordinate (Y1n) of the end point of the stroke of the “^”shape.

Further, each coordinate data may include time stamp information Tcorresponding to a time point at which a point corresponding to thiscoordinate data was handwritten. The time point at which the point washandwritten may be either an absolute time (e.g.year/month/date/hour/minute/second) or a relative time with reference toa certain time point. For example, an absolute time (e.g.year/month/date/hour/minute/second) at which a stroke began to behandwritten may be added as time stamp information to each stroke data,and furthermore a relative time indicative of a difference from theabsolute time may be added as time stamp information T to eachcoordinate data in the stroke data.

In this manner, by using the time-series information in which the timestamp information T is added to each coordinate data, the temporalrelationship between strokes can be more precisely expressed.

Moreover, information (Z) indicative of a pen stroke pressure may beadded to each coordinate data.

The time-series information (handwritten document information) 200having the structure as described with reference to FIG. 4 can expressnot only the trace of handwriting of each stroke, but also the temporalrelation between strokes. Thus, with the use of the time-seriesinformation 200, even if a distal end portion of the handwritten “arrow”is written over the handwritten character “A” or near the handwrittencharacter “A”, as shown in FIG. 3, the handwritten character “A” and thedistal end portion of the handwritten “arrow” can be treated asdifferent characters or graphics.

Furthermore, in the present embodiment, as described above, handwrittendocument information is stored not as an image or a result of characterrecognition, but as a set of time-series stroke data. Thus, handwrittencharacters can be handled, without depending on languages of thehandwritten characters. Therefore, the structure of the time-seriesinformation 200 of the present embodiment can be commonly used invarious countries of the world where different languages are used.

FIG. 5 illustrates an example of a note view screen 500 which isdisplayed on the touch-screen display 17 by a digital notebookapplication program of the tablet computer 10. The note view screen 500is a screen which enables new creation of handwritten data (handwrittenpage) and enables viewing and editing of an existing handwritten page.

The direction of the note view screen 500 is preset. The note viewscreen 500 is displayed on the touch-screen display 17 in such adirection that the upper side of the note view screen 500 is located onan upper end 21 side of the tablet computer 10 and the lower side of thenote view screen 500 is located on a lower end 23 side of the tabletcomputer 10. The direction of the note view screen 500 is fixed in theabove-described preset direction. Even if the direction of inclinationof the tablet computer 10 varies relative to the direction of gravity,the note view screen 500 is not rotated.

The note view screen 500 further displays a black pen button 511, a redpen button 512, a marker button 513, a select button 514, and an eraserbutton 515. The black pen button 511, red pen button 512 and markerbutton 513 are user interfaces for prompting the user to select the modeof drawing of strokes. The select button 514 is a button which is usedas the above-described range designation tool. The eraser button 515 isa button which is used as the above-described eraser tool.

For example, if a handwriting input operation with use of the pen 100 isexecuted on the note view screen 500 in the state in which the black penbutton 511 is selected by a tap gesture or the like by the user, thedigital notebook application program displays a black stroke (locus) onthe note view screen 500 in accordance with the movement of the pen 100.The user can handwrite an arbitrary character, mark, graphic or table onthe note view screen 500. Since the direction of the note view screen500 is fixed, as described above, the user can perform handwriting onthe note view screen 500 in a desired direction, by changing thedirection of inclination of the tablet computer 10 to an arbitrarydirection, that is, by setting the direction of the note view screen 500in an arbitrary direction.

FIG. 5 illustrates an example in which strokes 501 having atop-and-bottom direction agreeing with the top-and-bottom direction ofthe note view screen 500, and strokes 502 having a top-and-bottomdirection, which is opposite to the top-and-bottom direction of the noteview screen 500, are displayed.

The strokes 501 are a stroke series corresponding to a handwrittencharacter string “ABCDE”. The top-and-bottom direction of each stroke ofthe strokes 501 agrees with the top-and-bottom direction of the noteview screen 500 (the top-and-bottom direction of the tablet computer10). In other words, an upper end of each stroke of the strokes 501 islocated on the upper end 21 side of the tablet computer 10, and a lowerend of each stroke is located on the lower end 23 side of the tabletcomputer 10.

The strokes 502 are a stroke series corresponding to a handwrittencharacter string “FGHIJK”. The top-and-bottom direction of each of thestrokes 502 is opposite to the top-and-bottom direction of the note viewscreen 500 (the top-and-bottom direction of the tablet computer 10). Inother words, each stroke of the strokes 502 is vertically inverted. Anupper end of each stroke of the strokes 502 is located on the lower end23 side of the tablet computer 10, and a lower end of each stroke islocated on the upper end 21 side of the tablet computer 10.

The handwritten character string “ABCDE” corresponding to the strokes501 is a character string which is handwritten on the note view screen500 in the state in which the tablet computer 10 is inclined such thatthe top-and-bottom direction of the note view screen 500 agrees with thedirection of gravity, for example, as illustrated in FIG. 6. The state,in which the upper end 21 of the tablet computer 10 is located on theupper side of the lower end 23 in the direction of gravity, asillustrated in FIG. 6, corresponds to the inclination direction of thetablet computer 10 in which the top-and-bottom direction of the noteview screen 500 agrees with the direction of gravity.

The direction of inclination of the tablet computer 10 may be defined byusing various parameters. For example, a first line, which isperpendicular to both left and right sides of the touch-screen display17 (or the main body 11 of tablet computer 10), and a second line, whichis perpendicular to both upper and lower sides of the touch-screendisplay 17 (or the main body 11 of tablet computer 10), are defined. Adirection, in which either the first line or the second line extendsdownward relative to the direction of gravity, may be set to be thedirection of inclination of the tablet computer 10.

Each of strokes, which are handwritten while the table computer 10 is inthe state of FIG. 6, can be handled as a stroke having a directionagreeing with the direction of the note view screen 500. In other words,the top-and-bottom direction of each stroke, which is handwritten whilethe table computer 10 is in the state of FIG. 6, can be estimated to bea direction agreeing with the top-and-bottom direction of the note viewscreen 500.

The handwritten character string “FGHIJK” corresponding to the strokes502 is a character string which is handwritten on the note view screen500 in the state in which the tablet computer 10 is inclined such thatthe top-and-bottom direction of the note view screen 500 is opposite tothe direction of gravity, for example, as illustrated in FIG. 7. Thestate, in which the lower end 23 of the tablet computer 10 is located onthe upper side of the upper end 21 in the direction of gravity, asillustrated in FIG. 7, corresponds to the inclination direction of thetablet computer 10 in which the top-and-bottom direction of the noteview screen 500 opposite to the direction of gravity. Each of strokes,which are handwritten while the table computer 10 is in the state ofFIG. 7, can be handled as a stroke having a direction opposite to thedirection of the note view screen 500. In other words, thetop-and-bottom direction of each stroke, which is handwritten while thetable computer 10 is in the state of FIG. 7, can be estimated to be adirection opposite to the top-and-bottom direction of the note viewscreen 500.

As has been described above, in the present embodiment, since thedirection of the note view screen 500 is fixed, the user can performhandwriting on the note view screen 500 in a desired direction, bychanging the inclination direction of the tablet computer 10.Accordingly, the user can use the tablet computer 10 with the samefeeling as a real paper-based pocket notebook. In this case, however,since the relationship between the top-and-bottom direction of the noteview screen 500 (handwritten page) and the top-and-bottom direction ofeach stroke becomes unclear, there may be a case in which it isdifficult to exactly execute various processes on plural stroke data inthe handwritten page.

Taking this into account, in the embodiment, stroke directioninformation indicative of the top-and-bottom direction of each stroke isadded to the stroke data corresponding to each stroke. Thetop-and-bottom direction of a stroke may be determined with reference tothe direction of the note view screen 500. In this case, strokedirection information of a certain stroke is indicative of thetop-and-bottom direction of this stroke relative to the note view screen500.

FIG. 8 illustrates an example of the data structure of handwritten data(handwritten page) corresponding to one page.

The handwritten data includes “stroke ID”, “stroke direction(top-and-bottom)” and “stroke data” with respect to each stroke. The“stroke ID” is an identifier of each corresponding stroke. The “strokedirection (top-and-bottom)” is direction information which is indicativeof the top-and-bottom direction of each stroke relative to the screen(note view screen 500). The “stroke direction (top-and-bottom)” mayindicate either a first direction (“normal direction”) which agrees withthe top-and-bottom direction of the screen, or a second direction(“180°”) which is opposite to the first direction.

Alternatively, the “stroke direction (top-and-bottom)” may indicate oneof the first direction (“normal direction”) which agrees with thetop-and-bottom direction of the screen, the second direction (“180°”)which is opposite to the first direction, a third direction (“90°”) witha clockwise rotation of 90° relative to the top-and-bottom direction ofthe screen, and a fourth direction (“270°”) with a clockwise rotation of270° relative to the top-and-bottom direction of the screen.

FIG. 9 illustrates strokes 503 having the third direction (“90°”).

A handwritten character string “XYZ” corresponding to the strokes 503 isa character string which was handwritten on the note view screen 500 inthe state in which the tablet computer 10 is inclined such that thetop-and-bottom direction of the note view screen 500 is rotated by 90°to the left relative to the direction of gravity, for example, asillustrated in FIG. 9. The state, in which a right end 22 of the tabletcomputer 10 is located on the upper side of a left end 24 in thedirection of gravity, as illustrated in FIG. 9, corresponds to theinclination direction of the tablet computer 10 in which thetop-and-bottom direction of the note view screen 500 is rotated by 90°to the left relative to the direction of gravity. Each of strokes, whichare handwritten while the tablet computer 10 is in the state of FIG. 9,can be handled as a stroke having the third direction (“90°”). In otherwords, the top-and-bottom direction of each stroke, which is handwrittenwhile the table computer 10 is in the state of FIG. 9, can be estimatedto be a direction rotated by 90° relative to the top-and-bottomdirection of the note view screen 500.

FIG. 10 illustrates strokes 504 having the fourth direction (“270°”).

A handwritten character string “123456789” corresponding to the strokes504 is a character string which was handwritten on the note view screen500 in the state in which the tablet computer 10 is inclined such thatthe top-and-bottom direction of the note view screen 500 is rotated by90° to the right relative to the direction of gravity, for example, asillustrated in FIG. 10. The state, in which the left end 24 of thetablet computer 10 is located on the upper side of the right end 22 inthe direction of gravity, as illustrated in FIG. 10, corresponds to theinclination direction of the tablet computer 10 in which thetop-and-bottom direction of the note view screen 500 is rotated by 90°to the right relative to the direction of gravity. Each of strokes,which are handwritten while the tablet computer 10 is in the state ofFIG. 10, can be handled as a stroke having the fourth direction(“270°”). In other words, the top-and-bottom direction of each stroke,which is handwritten while the table computer 10 is in the state of FIG.10, can be estimated to be a direction rotated by 270° relative to thetop-and-bottom direction of the note view screen 500.

FIG. 11 shows an example of a handwritten data portion corresponding toa handwritten character “A” in the strokes 501 of FIG. 5. Thehandwritten character “A” includes a stroke ST1 and a stroke ST2.

As regards the stroke ST1, the stroke ID=1 is an identifier of thestroke ST1. The stroke direction=“normal direction” of the stroke ST1indicates that the top-and-bottom direction of the stroke ST1 is theabove-described “normal direction”, that is, the top-and-bottomdirection of the stroke ST1 agrees with the direction of the screen. Thestroke data of the stroke ST1 is indicative of a plurality ofcoordinates of a plurality of points on the stroke ST1.

As regards the stroke ST2, the stroke ID=2 is an identifier of thestroke ST2. The stroke direction=“normal direction” of the stroke ST2indicates that the top-and-bottom direction of the stroke ST2 is theabove-described “normal direction”, that is, the top-and-bottomdirection of the stroke ST2 agrees with the direction of the screen. Thestroke data of the stroke ST2 is indicative of a plurality ofcoordinates of a plurality of points on the stroke ST2.

FIG. 12 shows an example of a handwritten data portion corresponding toa handwritten character “F” in the strokes 502 of FIG. 5 (in FIG. 5, “F”is read as an inverted F). The handwritten character “F” includes astroke ST13, a stroke ST14 and a stroke ST15.

As regards the stroke ST13, the stroke ID=13 is an identifier of thestroke ST13. The stroke direction=“180°” of the stroke ST13 indicatesthat the top-and-bottom direction of the stroke ST13 is theabove-described “180°”, that is, the top-and-bottom direction of thestroke ST13 is opposite to the direction of the screen. The stroke dataof the stroke ST13 is indicative of a plurality of coordinates of aplurality of points on the stroke ST13.

As regards the stroke ST14, the stroke ID=14 is an identifier of thestroke ST14. The stroke direction=“180°” of the stroke ST14 indicatesthat the top-and-bottom direction of the stroke ST14 is theabove-described “180°”, that is, the top-and-bottom direction of thestroke ST14 is opposite to the direction of the screen. The stroke dataof the stroke ST14 is indicative of a plurality of coordinates of aplurality of points on the stroke ST14.

As regards the stroke ST15, the stroke ID=15 is an identifier of thestroke ST15. The stroke direction=“180°” of the stroke ST15 indicatesthat the top-and-bottom direction of the stroke ST15 is theabove-described “180°”, that is, the top-and-bottom direction of thestroke ST15 is opposite to the direction of the screen. The stroke dataof the stroke ST15 is indicative of a plurality of coordinates of aplurality of points on the stroke ST15.

In the embodiment, the top-and-bottom direction of each stroke isdetermined by using at least either the inclination direction of thetablet computer 10 which is determined in accordance with an output ofan acceleration sensor in the tablet computer 10, or the positionalrelationship between the tablet computer 10 and the user, which isdetermined in accordance with a result of face recognition using animage captured by the camera module 18.

FIG. 13 illustrates an example of the directions of three axes of theacceleration sensor.

An X axis is, for example, an axis which is parallel to short sides (aleft side and a right side) of the touch-screen display 14. The frontside of the X axis is set at +X and the back side of the X axis is setat −X. A Y axis is, for example, an axis which is parallel to long sides(an upper side and a lower side) of the touch-screen display 14. Theleft side of the Y axis is set at −Y and the right side of the Y axis isset at +Y. A Z axis is, for example, an axis which is perpendicular tothe screen of the touch-screen display (the top surface of the tabletcomputer 10). The top end side of the Z axis is set at −Z and the bottomend side of the Z axis is set at +Z.

In the state in which the tablet computer 10 is placed on a horizontalplane such as on the desk, as illustrated in FIG. 13, the output (x, y,z) of the acceleration sensor is (0, 0, 1).

FIG. 14 illustrates a relationship between the inclination direction ofthe tablet computer 10 (the direction of the tablet computer 10) and adetection output of the acceleration sensor. In order to simplify thedescription, it is assumed that the back surface of the tablet computer10 is substantially vertical to the horizontal axis.

Part (A) of FIG. 14 shows an inclination direction of the tabletcomputer 10 in which the direction of strokes which are input byhandwriting may be determined to be “normal direction”. The output (x,y, z) of the acceleration sensor is (1, 0, 0).

Part (B) of FIG. 14 shows an inclination direction of the tabletcomputer 10 in which the direction of strokes which are input byhandwriting may be determined to be “180°”. The output (x, y, z) of theacceleration sensor is (−1, 0, 0).

Part (C) of FIG. 14 shows an inclination direction of the tabletcomputer 10 in which the direction of strokes which are input byhandwriting may be determined to be “90°”. The output (x, y, z) of theacceleration sensor is (0, −1, 0).

Part (D) of FIG. 14 shows an inclination direction of the tabletcomputer 10 in which the direction of strokes which are input byhandwriting may be determined to be “270°”. The output (x, y, z) of theacceleration sensor is (0, 1, 0).

In this manner, by using the direction of inclination of the tabletcomputer 10 which is determined by the output of the accelerationsensor, the top-and-bottom direction of each of strokes, which are inputby handwriting, can be determined.

Alternatively, the top-and-bottom direction of each of strokes, whichare input by handwriting, may be determined by using the positionalrelationship between the tablet computer 10 and the user, which isdetermined in accordance with a result of face recognition using animage captured by the camera module 18, instead of the inclinationdirection of the tablet computer 10 which is determined by the output ofthe acceleration sensor.

The positional relationship between the tablet computer 10 and the usercan be estimated by determining whether a face image that is recognizedis, for example, a face image of the face as viewed in a frontaldirection, a face image which is vertically inverted, a face imagerotated by 90° to the right, or a face image rotated by 270° to theright.

Alternatively, the top-and-bottom direction of each of strokes, whichare input by handwriting, may be determined by selectively using,depending on the condition, the inclination direction of the tabletcomputer 10 which is determined by the output of the accelerationsensor, or the positional relationship between the tablet computer 10and the user, which is determined in accordance with a result of facerecognition.

In the meantime, when the tablet computer 10 is used in the state inwhich the tablet compute 10 is placed on the horizontal plane, as shownin FIG. 13, there is substantially no inclination of the tablet computer10.

Thus, in the method of determining the direction of a stroke (thetop-and-bottom direction of a stroke) by using only the output of theacceleration sensor, if the tablet computer 10 is rotated by 180° on thehorizontal plane by the user such that the upper end 21 of the tabletcomputer 10 is directed to the user, and if the user performs ahandwriting operation from the upper end 21 side of the tablet computer10, it is possible that the direction of a stroke (the top-and-bottomdirection of a stroke) which is input by the handwriting operationcannot correctly be determined.

In addition, there is a case in which the user uses the tablet computer10 by placing it on the user's laps. In this case, the attitude of thetablet computer 10 becomes unstable. Thus, in the method of determiningthe direction of a stroke (the top-and-bottom direction of a stroke) byusing only the output of the acceleration sensor, it is possible, forexample, that the top-and-bottom direction of a stroke, relative to thescreen, is erroneously determined to be “90°”, “180°” or “270°”, despitethe user intending to handwrite the stroke in the “normal direction”.

In some cases, such erroneous determination can be prevented by themethod of determining the direction of a stroke by using a facerecognition result.

FIG. 15 illustrates an example of an operation of determining strokedirection information with use of a face recognition result.

In FIG. 15, the case is assumed that the user is using the tabletcomputer 10 while viewing the screen (note view screen 500) of thetablet computer 10 in the normal direction. The use mode of FIG. 15corresponds to, for example, the state in which the tablet computer 10is placed on the horizontal plane and the lower end side of the tabletcomputer 10 is directed to the user, or the state in which the userholds the tablet computer 10 such that note view screen 500 is set inthe normal direction. The digital notebook application program executesa face recognition process 700 for detecting a face image of a person(user) included in an image 600 which is captured by the camera module18. Based on the result of the face recognition process 700, the digitalnotebook application program can specify the direction of the face imagein the image 600. Accordingly, based on the result of the facerecognition process 700, the digital notebook application program canrecognize that the lower end side of the tablet computer 10 is directedto the user, that is, the positional relationship between the tabletcomputer 10 and the user is such a positional relationship that a strokecan be handwritten in the above-described “normal direction”.

FIG. 16 illustrates another example of the operation of determiningstroke direction information with use of a face recognition result.

In FIG. 16, the case is assumed that the user is using the tabletcomputer 10 while viewing the screen (note view screen 500) of thetablet computer 10 which is vertically inverted. The use mode of FIG. 16corresponds to, for example, the state in which the tablet computer 10is placed on the horizontal plane and the upper end side of the tabletcomputer 10 is directed to the user, or the state in which the userholds the tablet computer 10 such that note view screen 500 isvertically inverted. The digital notebook application program executesthe face recognition process 700 for detecting a face image of a person(user) included in an image 601 which is captured by the camera module18. Based on the result of the face recognition process 700, the digitalnotebook application program can specify the direction of the face imagein the image 601. Accordingly, based on the result of the facerecognition process 700, the digital notebook application program canrecognize that the upper end side of the tablet computer 10 is directedto the user, that is, the positional relationship between the tabletcomputer 10 and the user is such a positional relationship that a strokecan be handwritten in the direction of the above-described “180°”.

FIG. 17 illustrates another example of the operation of determiningstroke direction information with use of a face recognition result.

In FIG. 17, the case is assumed that the user is using the tabletcomputer 10 while viewing the screen (note view screen 500) of thetablet computer 10 which is rotated by 90° to the right. The use mode ofFIG. 17 corresponds to, for example, the state in which the tabletcomputer 10 is placed on the horizontal plane and the right end side ofthe tablet computer 10 is directed to the user, or the state in whichthe user holds the tablet computer 10 such that note view screen 500 isrotated by 90° to the right. The digital notebook application programexecutes the face recognition process 700 for detecting a face image ofa person (user) included in an image 602 which is captured by the cameramodule 18. Based on the result of the face recognition process 700, thedigital notebook application program can specify the direction of theface image in the image 602. Accordingly, based on the result of theface recognition process 700, the digital notebook application programcan recognize that the right end side of the tablet computer 10 isdirected to the user, that is, the positional relationship between thetablet computer 10 and the user is such a positional relationship that astroke can be handwritten in the direction of the above-described“270°”.

FIG. 18 illustrates another example of the operation of determiningstroke direction information with use of a face recognition result.

In FIG. 18, the case is assumed that the user is using the tabletcomputer 10 while viewing the screen (note view screen 500) of thetablet computer 10 which is rotated by 90° to the left. The use mode ofFIG. 18 corresponds to, for example, the state in which the tabletcomputer 10 is placed on the horizontal plane and the left end side ofthe tablet computer 10 is directed to the user, or the state in whichthe user holds the tablet computer 10 such that note view screen 500 isrotated by 90° to the left. The digital notebook application programexecutes the face recognition process 700 for detecting a face image ofa person (user) included in an image 603 which is captured by the cameramodule 18. Based on the result of the face recognition process 700, thedigital notebook application program can specify the direction of theface image in the image 603. Accordingly, based on the result of theface recognition process 700, the digital notebook application programcan recognize that the left end side of the tablet computer 10 isdirected to the user, that is, the positional relationship between thetablet computer 10 and the user is such a positional relationship that astroke can be handwritten in the direction of the above-described “90°”.

In the meantime, the top-and-bottom direction of a handwrittencharacter, etc. can be determined based on a direction of writing of ahandwritten character. However, in the method of determining thetop-and-bottom direction, based on the direction of writing, it isdifficult to exactly determine the top-and-bottom direction because ofcharacteristics of this method, and this method cannot be adaptive toboth LTR language and RTL language.

In the embodiment, since the top-and-bottom direction of each stroke isdetermined by using at least either the output of the accelerationsensor or the face recognition result, the top-and-bottom direction canexactly be determined, no matter whether the language of the handwrittencharacter is LTR language or RTL language.

FIG. 19 shows a system configuration of the tablet computer 10.

As shown in FIG. 19, the tablet computer 10 includes, in addition to theabove-described camera module 18, a CPU 101, a system controller 102, amain memory 103, a graphics controller 104, a BIOS-ROM 105, anonvolatile memory 106, a wireless communication device 107, anacceleration sensor 108, and an embedded controller (EC) 109.

The CPU 101 is a processor which controls the operations of variousmodules in the tablet computer 10. The CPU 101 executes various kinds ofsoftware, which are loaded from the nonvolatile memory 106 that is astorage device into the main memory 103. The software includes anoperating system (OS) 201 and various application programs. Theapplication programs include a digital notebook application program 202.The digital notebook application program 202 includes a function ofcreating and displaying the above-described handwritten data, a functionof editing the handwritten data, a handwriting search function, and arecognition function.

In addition, the CPU 101 executes a basic input/output system (BIOS)which is stored in the BIOS-ROM 105. The BIOS is a program for hardwarecontrol.

The system controller 102 is a device which connects a local bus of theCPU 101 and various components. The system controller 102 includes amemory controller which access-controls the main memory 103. Inaddition, the system controller 102 includes a function of communicatingwith the graphics controller 104 via, e.g. a PCI EXPRESS serial bus.

The graphics controller 104 is a display controller which controls anLCD 17A that is used as a display monitor of the tablet computer 10. Adisplay signal, which is generated by the graphics controller 104, issent to the LCD 17A. The LCD 17A displays a screen image based on thedisplay signal. A touch panel 17B and a digitizer 17C are disposed onthe LCD 17A. The touch panel 17B is an electrostatic capacitance-typepointing device for executing an input on the screen of the LCD 17A. Acontact position on the screen, which is touched by a finger, and amovement of the contact position are detected by the touch panel 17B.The digitizer 17C is an electromagnetic induction-type pointing devicefor executing an input on the screen of the LCD 17A. A contact positionon the screen, which is touched by the pen 100, and a movement of thecontact position are detected by the digitizer 17C.

The wireless communication device 107 is a device configured to executewireless communication such as wireless LAN or 3G mobile communication.The acceleration sensor 108 is, for example, a three-dimensionalacceleration sensor, and is configured to detect a dynamic/staticacceleration corresponding to each of the three axial directions. The EC109 is a one-chip microcomputer including an embedded controller forpower management. The EC 109 includes a function of powering on orpowering off the tablet computer 10 in accordance with an operation of apower button by the user.

Next, referring to FIG. 20, a description is given of a functionalconfiguration of the digital notebook application program 202.

The digital notebook application program 202 includes a pen locusdisplay process module 301, a time-series information generator 302, anedit process module 303, a page storage process module 304, a pageacquisition process module 305, a handwritten document display processmodule 306, a process-target block select module 307, a process module308, and a stroke up-and-down direction determination module 312.

The digital notebook application program 202 executes creation, displayand edit of a handwritten document (handwritten data) by using strokedata which is input by using the touch-screen display 17. Thetouch-screen display 17 is configured to detect the occurrence of eventssuch as “touch”, “move (slide)” and “release”. The “touch” is an eventindicating that an external object has come in contact with the screen.The “move (slide)” is an event indicating that the position of contactof the external object has been moved while the external object is incontact with the screen. The “release” is an event indicating that theexternal object has been released from the screen.

The pen locus display process module 301 and time-series informationgenerator 302 receive an event of “touch” or “move (slide)” which isgenerated by the touch-screen display 17, thereby detecting ahandwriting input operation. The “touch” event includes coordinates of acontact position. The “move (slide)” event also includes coordinates ofa contact position at a destination of movement. Thus, the pen locusdisplay process module 301 and time-series information generator 302 canreceive coordinate series, which correspond to the locus of movement ofthe contact position, from the touch-screen display 17.

The pen locus display process module 301 receives coordinate series fromthe touch-screen display 17 and displays, based on the coordinateseries, the loci of plural strokes, which are input by a handwritinginput operation with use of the pen 100 or the like, on the screen ofthe LCD 17A in the touch-screen display 17. By the pen locus displayprocess module 301, the locus of the pen 100 during a time in which thepen 100 is in contact with the screen, that is, the locus of eachstroke, is drawn on the screen of the LCD 17A.

The stroke up-and-down direction determination module 312 determines atop-and-bottom direction for each of plural strokes. To be morespecific, the stroke up-and-down direction determination module 312determines a top-and-bottom direction for each of plural strokes whichhave been input by handwriting, by using at least either (1) theinclination direction of the tablet computer 10 which is determined inaccordance with an output of the acceleration sensor 108, or (2) thepositional relationship between the tablet computer 10 and the user,which is determined in accordance with a result of face recognitionusing an image captured by the camera module 18. The top-and-bottomdirections of strokes may be the top-and-bottom directions of strokesrelative to the screen, as described above.

The time-series information generator 302 receives the above-describedcoordinate series which are output from the touch-screen display 17, andgenerates, based on the coordinate series, a plurality of stroke data(time-series information) corresponding to the above-described pluralstrokes. The stroke data (time-series information), that is, thecoordinates corresponding to the respective points of each stroke andthe time stamp information of each stroke, may be temporarily stored ina working memory 401.

The page storage process module 304 adds the above-described strokedirection information to each stroke data, based on the up-and-downdirection of each stroke which is determined by the stroke up-and-downdirection determination module 312. Then, the page storage processmodule 304 stores in a storage medium 402 handwritten data (handwrittenpage), the handwritten data including plural stroke data (time-seriesinformation) corresponding to plural strokes and the plural pieces ofstroke direction information indicative of the top-and-bottom directionsof the plural strokes relative to the screen. The storage medium 402, asdescribed above, may be the storage device in the tablet computer 10,the storage device in the personal computer 1, or the storage device inthe server 2.

The page acquisition process module 305 reads out from the storagemedium 402 arbitrary handwritten data which is already stored in thestorage medium 402. The read-out handwritten data is sent to thehandwritten document display process module 306. The handwrittendocument display process module 306 analyzes the handwritten data anddisplays, based on the analysis result, the loci of strokes indicated bythe handwritten data on the screen as a handwritten page.

The edit process module 303 executes a process for editing a handwrittenpage which is currently being displayed. Specifically, in accordancewith an edit operation which is executed by the user on the touch-screendisplay 17, the edit process module 303 executes an edit process fordeleting or moving one or more strokes of a plurality of stokes whichare being displayed. Further, the edit process module 303 updates thetime-series information which is being displayed, in order to reflectthe result of the edit process on the time-series information.

The user can delete an arbitrary stroke of the plural strokes which arebeing displayed, by using an “eraser” tool, etc. In addition, the usercan designate a range of an arbitrary part in the handwritten page whichis being displayed, by using a “range designation” tool for surroundingan arbitrary part on the screen by a circle or a rectangle. Inaccordance with the designated range on the screen, which is designatedby this range designation operation, a handwritten data part that is thetarget of processing, that is, a set of strokes that are the target ofprocessing, is selected by the process-target block select module 307.

When a menu such as “delete” or “move” has been selected from the editmenu by the user, the edit process module 303 executes a process ofdelete or move on the set of stroke data which has been selected by theprocess-target block select module 307.

The process module 308 can execute various processes, for example, ahandwriting search process and a recognition process, on theprocess-target handwritten data. The process module 308 includes asearch process module 309 and a recognition process module 310.

The search process module 309 executes a handwriting search (strokesearch). This handwriting search is a process of searching for ahandwritten page including at least one stroke similar to at least onestroke (query stroke), which is a search key, from the storage medium402. In addition, in this handwriting search, it is possible to searchfor at least one stroke similar to at least one query stroke from asearch-target handwritten page.

In the handwriting search, a stroke similar to a query stroke issearched from plural strokes by matching between the query stroke andthese plural strokes in the search-target handwritten page. The strokesimilar to the query stroke is a stroke having a handwritingcharacteristic similar to the handwriting characteristic of the querystroke. The handwriting characteristic is, for example, the shape of thestroke, the direction of writing of the stroke, etc.

Various methods are usable as the method of calculating the degree ofsimilarity between strokes. For example, coordinate series of eachstroke may be treated as a vector. In this case, in order to calculatethe degree of similarity between vectors which are targets ofcomparison, an inner product between the vectors which are targets ofcomparison may be calculated. In many cases, a search key includes aplurality of strokes. In such cases, a handwritten page including astroke series similar to plural strokes (query stroke series), which isa search key, is searched from the storage medium 402. In the search forthe query stroke series, DP (Dynamic Programming) matching may be used.

The recognition process module 310 executes a recognition process, suchas character recognition, on a handwritten page that is being displayed.The case is now assumed that the entirety of a certain handwritten pageor a part in this handwritten page is subject to character recognition.It is possible that the entirety of the handwritten page or aprocess-target part in this handwritten page includes, for example, aplurality of strokes having the above-described direction of “180°”. Inthis case, the recognition process module 310 extracts the pluralstrokes having the direction of “180°” from the handwritten page or theprocess-target part in this handwritten page, and converts, bycoordinate conversion, these plural strokes to other plural strokeshaving a direction corresponding to the “normal direction”. Then, therecognition process module 310 executes character recognition of theseother plural strokes, and outputs at least one character codecorresponding to these other plural strokes. Thereby, even if theentirety of a handwritten page or a process-target part in thishandwritten page includes strokes having the direction of “180°”,character recognition can correctly be executed on these strokes.

FIG. 21 illustrates a character recognition process operation on strokes501 corresponding to a character string “ABCDE” having a direction of“normal direction”. Based on the above-described stroke directioninformation, the recognition process module 310 determines whether thestrokes 501 are strokes having the direction of “normal direction”.Since the strokes 501 are strokes having the direction of “normaldirection”, the recognition process module 310 does not execute theabove-described coordinate conversion. Then, using a characterrecognition module in the recognition process module 310, therecognition process module 310 executes character recognition forconverting the strokes 501 to at least one character code (in thisexample, a plurality of character codes corresponding to the characterstring “ABCDE”).

FIG. 22 illustrates a character recognition process operation on strokes502 corresponding to a character string “FGHIJK” having the direction of“180°”. Based on the above-described stroke direction information, therecognition process module 310 determines whether the strokes 502 arestrokes having the direction of “normal direction”. The strokes 502 arenot strokes having the direction of “normal direction”, but are strokeshaving the direction of “180°”. Thus, the recognition process module 310executes the above-described coordinate conversion (180°rotation/inversion), and converts the strokes 502 to strokes 502′ havingthe direction of “normal direction”. Then, using the characterrecognition module in the recognition process module 310, therecognition process module 310 executes character recognition forconverting the strokes 502′ to at least one character code (in thisexample, a plurality of character codes corresponding to the characterstring “FGHIJK”).

In the meantime, also in the case of character-recognizing strokeshaving the direction of “90°”, coordinate conversion is executed forconverting the strokes having the direction of “90°” to strokes havingthe direction of “normal direction”. Similarly, in the case ofcharacter-recognizing strokes having the direction of “270°”, coordinateconversion is executed for converting the strokes having the directionof “270°” to strokes having the direction of “normal direction”.

FIG. 23 illustrates the outline of a handwriting search.

The search process module 309 displays on the screen a search key inputdialog including a search key input area 521 and a search button 522.The search key input area 521 is an input area for handwriting acharacter string, a graphic, a table, etc., which is to be set as asearch key. The search button 522 is a button for requesting executionof a search process. FIG. 23 illustrates the case in which a handwrittencharacter string “Meeting” has been input as a search key in the searchkey input area 521. The user can handwrite a handwritten graphic, ahandwritten table, etc., as well as the handwritten character string, inthe search key input area 521.

If it is detected that the search button 522 has been tapped, the searchprocess module 309 determines the strokes, which correspond to thehandwritten character string “Meeting”, to be a query stroke series, andsearches for a handwritten page including a stroke series similar tothis query stroke series. The case is now assumed that handwritten pages611, 612 and 613 include strokes corresponding to the handwrittencharacter string “Meeting”. In this case, the search process module 309displays thumbnails of the handwritten pages 611, 612 and 613 on thescreen. Further, the search process module 309 may display, withemphasis, hit words (handwritten character string “Meeting”) in thehandwritten pages 611, 612 and 613.

It is possible that a search-target handwritten page includes aplurality of strokes having the above-described direction of “180°”. Inthis case, the search process module 309 extracts the strokes having thedirection of “180°” from the handwritten page, and converts, bycoordinate conversion, these strokes to other plural strokes having thedirection corresponding to “normal direction”. Then, by matching betweenthe query strokes and these other plural strokes, the search processmodule 309 searches for strokes similar to the query strokes from theseother plural strokes. Thereby, even if a plurality of strokes having thedirection of “180°” are included in a handwritten page, a handwritingsearch for these strokes can correctly be executed.

FIG. 24 illustrates an example of the handwriting search. The case isnow assumed that a handwritten page including strokes 501 and 502, whichhave been described with reference to FIG. 5, is a search target. It isassumed that the query stroke series is a handwritten character string“ABC”.

Based on the above-described stroke direction information, the searchprocess module 309 determines whether the strokes 501 are strokes havingthe direction of “normal direction”. Since the strokes 501 are strokeshaving the direction of “normal direction”, the search process module309 does not execute the above-described coordinate conversion. Inaddition, based on the above-described stroke direction information, thesearch process module 309 determines whether the strokes 502 are strokeshaving the direction of “normal direction”. Since the strokes 502 arestrokes having the direction of “180°”, the search process module 309executes the above-described coordinate conversion (180°rotation/inversion), and converts the strokes 502 to strokes 502′ havingthe direction of “normal direction”. Then, the search process module 309searches for strokes similar to the query strokes from the strokes 501and strokes 502′.

In FIG. 24, the handwritten character string “ABC” in the strokes 501 issearched as the stroke series similar to the query stroke series, andthis handwritten character string “ABC” is displayed with emphasis.

FIG. 25 illustrates a handwriting search operation which is executedwhen a handwritten character string “FGH”, instead of the handwrittencharacter string “ABC”, has been input as the query stroke series. InFIG. 25, by matching between the strokes 502′ and the query strokeseries, the handwritten character string “FGH” in the strokes 502′ issearched as the stroke series similar to the query stroke series. Then,the handwritten character string “FGH” in the strokes 502 is displayedwith emphasis.

In the meantime, also when strokes having the direction of “90°” areincluded in a search-target page, coordinate conversion is executed forconverting the strokes having the direction of “90°” to strokes havingthe direction of “normal direction”. Similarly, when strokes having thedirection of “270°” are included in a search-target page, coordinateconversion is executed for converting the strokes having the directionof “270°” to strokes having the direction of “normal direction”.

FIG. 26 illustrates an example of a handwritten page including bothstrokes having the direction of “normal direction” and strokes havingthe direction opposite to “normal direction”. This handwritten pageincludes strokes 801 having the direction of “normal direction” andstrokes 802 having the direction of “180°”. The handwritten documentdisplay process module 306 can selectively display either of the strokes801 and 802, which have different directions, on the note view screen500.

For example, the handwritten document display process module 306 canextract only stroke data corresponding to the strokes 801 having thedirection of “normal direction” from the handwritten page, and candisplay only the strokes 801 having the direction of “normal direction”on the note view screen 500. Alternatively, the handwritten documentdisplay process module 306 can display either the strokes 801 having thedirection of “normal direction” or the strokes 802 having the directionof “180°” on the note view screen 500 with emphasis.

Besides, the handwritten document display process module 306 canselectively display either the strokes 801 or the strokes 802 on thenote view screen 500, in accordance with the present inclinationdirection of the tablet computer 10.

FIG. 27 illustrates an example of the operation of selectivelydisplaying either strokes having a direction corresponding to the normaldirection or strokes having a direction opposite to the normaldirection, in accordance with the present inclination direction of thetablet computer 10. The case is now assumed that the present inclinationdirection of the tablet computer 10 is such an inclination directionthat the direction of the note view screen 500 agrees with the directionof gravity. In this case, the loci of the strokes 801, which areobserved by the user, are in the normal state in which these loci arenot vertically inverted. However, the loci of the strokes 802, which areobserved by the user, are in the vertically inverted state. Accordingly,in this case, the handwritten document display process module 306 maydisplay only the strokes 801 having the direction agreeing with thedirection of the note view screen 500. Alternatively, the handwrittendocument display process module 306 may display with emphasis thestrokes 801 having the direction agreeing with the direction of the noteview screen 500, and may display the strokes 802 in a thin color.

Furthermore, the handwritten document display process module 306 canalso vertically invert the strokes 802, and can display the verticallyinverted strokes 802 on the note view screen 500. Thereby, the strokes801 and the strokes 802 can be display on the note view screen 500 inthe direction agreeing with the direction of the note view screen 500.

As shown in FIG. 28, if the present inclination direction of the tabletcomputer 10 is such a direction that the direction of the note viewscreen 500 is opposite to the direction of gravity, the handwrittendocument display process module 306 may display only the strokes 802.Alternatively, the handwritten document display process module 306 maydisplay with emphasis the strokes 802, and may display the strokes 801in a thin color.

A flowchart of FIG. 29 illustrates the procedure of a handwritten datastorage process.

To start with, the stroke up-and-down direction determination module 312determines the inclination direction of the tablet computer 10, based onthe detection output of the acceleration sensor 108 (step S11). Next,the stroke up-and-down direction determination module 312 executes facerecognition for detecting a face image of a person (user) included in animage which is captured by the camera module 18, and estimates, based onthe result of the face recognition, the present positional relationshipbetween the tablet computer 10 and the user (step S12). Then, the strokeup-and-down direction determination module 312 determines thetop-and-bottom directions of strokes which have been input byhandwriting, by using at least either the present inclination directionof the tablet computer 10, or the present positional relationshipbetween the tablet computer 10 and the user (step S13).

In step S13, the stroke up-and-down direction determination module 312may determine, for example, whether the present inclination of thetablet computer is less than a reference value. When the tablet computer10 is used in a state in which the back surface thereof is substantiallyhorizontal to the horizontal plane, the present inclination of thetablet computer is less than the reference value. In this case, thestroke up-and-down direction determination module 312 determines thetop-and-bottom directions of strokes which have been input byhandwriting, by using the present positional relationship between thetablet computer 10 and the user, in preference to the presentinclination direction of the tablet computer 10 which is determined bythe output of the acceleration sensor 108. For example, if the presentpositional relationship between the tablet computer 10 and the user hassuccessfully been determined by the above-described face recognition,the top-and-bottom directions of strokes relative to the screen may bedetermined based on only the present positional relationship. Thereby,the top-and-bottom directions can be determined more exactly than in thecase of using the present inclination direction of the tablet computer10 which is determined by the output of the acceleration sensor 108.

On the other hand, when the present inclination of the tablet computeris the reference value or more, the stroke up-and-down directiondetermination module 312 may determine the top-and-bottom directions ofstrokes which have been input by handwriting, by using the presentinclination direction of the tablet computer 10, in preference to thepresent positional relationship between the tablet computer 10 and theuser. In this case, the stroke up-and-down direction determinationmodule 312 may determine the top-and-bottom directions of strokesrelative to the screen, based on only the present inclination directionof the tablet computer 10. Thereby, the top-and-bottom directions ofstrokes can be determined more quickly with a lighter load than in thecase of using the face recognition result.

The page storage process module 304 stores in the storage medium thehandwritten data, the handwritten data including stroke datacorresponding to strokes which are input by handwriting, and strokedirection information indicative of the top-and-bottom directions of thestrokes which are input by handwriting (step S14).

In the meantime, as described above, the top-and-down directions of thestrokes which are input by handwriting may be determined by using onlythe present inclination direction of the tablet computer 10, or only thepresent positional relationship between the tablet computer 10 and theuser.

FIG. 30 illustrates the procedure of a process of displaying a pluralityof strokes, based on handwritten data.

The handwritten document display process module 306 reads outhandwritten data, which is a display process target, from the storagemedium. Then, based on the stroke direction information that is added toeach stroke data in the handwritten data, the handwritten documentdisplay process module 306 classifies plural stroke data in thehandwritten data into a plurality of groups (step S21). In step S21, theplural strokes are classified into four groups, namely a group ofstrokes having a direction of “normal direction”, a group of strokeshaving a direction of “90°”, a group of strokes having a direction of“180°”, and a group of strokes having a direction of “270°”. Then, thehandwritten document display process module 306 selectively displays theplural groups on the screen (step S22). In step S22, for example, thehandwritten document display process module 306 may select, from thefour groups, a group having the top-and-bottom direction agreeing withthe present inclination direction of the tablet computer 10, and maydraw the strokes belonging to the selected group on the screen.

As has been described above, in the embodiment, the handwritten data,which includes a plurality of stroke data corresponding to a pluralityof strokes and a plurality of pieces of direction information relatingto the top-and-bottom directions of the plurality of strokes, is storedin the storage medium. Therefore, the individual stroke data can beprocessed in the state in which their top-and-bottom directions aretaken into account. In addition, the top-and-bottom directions of theplural strokes are determined by using at least either the inclinationdirection of the tablet computer 10 which is determined in accordancewith the output of the acceleration sensor 108, or the positionalrelationship between the tablet computer 10 and the user, which isdetermined in accordance with the result of face recognition using animage captured by the camera module 18. Therefore, the top-and-bottomdirections can be determined more exactly than in the case ofdetermining the top-and-bottom directions by using the direction ofwriting of strokes.

Since the various processes on handwritten documents in the embodimentcan be realized by a computer program, the same advantageous effects aswith the present embodiment can easily be obtained simply by installingthe computer program into an ordinary computer with a touch-screendisplay through a computer-readable storage medium which stores thecomputer program, and executing the computer program.

The various modules of the systems described herein can be implementedas software applications, hardware and/or software modules, orcomponents on one or more computers, such as servers. While the variousmodules are illustrated separately, they may share some or all of thesame underlying logic or code.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the inventions.

What is claimed is:
 1. An electronic device comprising: a displayprocessor configured to display on a screen a plurality of strokes inputby handwriting; a hardware processor configured to determine atop-and-bottom direction for each of the plurality of strokes; and astorage processor configured to store in a storage medium handwrittendata, the handwritten data comprising a plurality of stroke datacorresponding to the plurality of strokes, and a plurality of pieces ofdirection information relating to a top-and-bottom direction of each ofthe plurality of strokes, wherein the hardware processor is configuredto determine a top-and-bottom direction of each of the plurality ofstrokes, by using at least either a direction of inclination of thedevice which is determined in accordance with an output of anacceleration sensor in the device, or a positional relationship betweenthe device and a user which is determined in accordance with a result offace recognition using an image captured by a camera module of thedevice; wherein the display processor is configured to select strokedata having a top-and-bottom direction agreeing with a present directionof inclination of the device, from between at least one first strokedata having a top-and-bottom direction which is a first direction, andat least one second stroke data having a top-and-bottom direction whichis a second direction that is different from the first direction, and todisplay at least one stroke corresponding to the selected stroke data onthe screen.
 2. The electronic device of claim 1, wherein the positionalrelationship between the device and the user, which is determined inaccordance with the result of the face recognition, is configured todetermine a top-and-bottom direction of each of the plurality ofstrokes, when the inclination of the device is less than a firstinclination.
 3. The electronic device of claim 1, wherein the directionof inclination of the device is configured to determine a top-and-bottomdirection of each of the plurality of strokes, when the inclination ofthe device is the first inclination or more.
 4. The electronic device ofclaim 1, wherein each of the plurality of pieces of directioninformation is indicative of either a first direction which agrees witha top-and-bottom direction of the screen, or a second direction oppositeto the first direction.
 5. The electronic device of claim 1, whereineach of the plurality of pieces of direction information is indicativeof any one of a first direction which agrees with a top-and-bottomdirection of the screen, a second direction opposite to the firstdirection, a third direction which is rotated by 90° relative to thetop-and-bottom direction of the screen, and a fourth direction which isrotated by 270° relative to the top-and-bottom direction of the screen.6. The electronic device of claim 1, wherein the display processor isconfigured to selectively display on the screen at least one strokecorresponding to at least one first stroke data having a top-and-bottomdirection which is a first direction, and at least one strokecorresponding to at least one second stroke data having a top-and-bottomdirection which is a second direction that is different from the firstdirection.
 7. The electronic device of claim 1, further comprising aconverter configured to convert, by coordinate conversion, a pluralityof first strokes having a top-and-bottom direction which is a seconddirection that is different from a first direction agreeing with adirection of the screen, to a plurality of second strokes having atop-and-bottom direction which is the first direction, and to output acharacter code corresponding to the plurality of second strokes.
 8. Theelectronic device of claim 1, further comprising a search processorconfigured to convert, by coordinate conversion, a plurality of firststrokes having a top-and-bottom direction which is a second directionthat is different from a first direction agreeing with a direction ofthe screen, to a plurality of second strokes having a top-and-bottomdirection which is the first direction, and to search for a strokesimilar to a third stroke which is a search key from the plurality offirst strokes, by matching between the third stroke which is the searchkey and the plurality of second strokes.
 9. A method of processinghandwritten data, comprising: displaying on a screen of an electronicdevice a plurality of strokes input by handwriting, the electronicdevice comprising a display processor; determining, by a hardwareprocessor, a top-and-bottom direction for each of the plurality ofstrokes; and storing in a storage medium handwritten data, thehandwritten data comprising a plurality of stroke data corresponding tothe plurality of strokes, and a plurality of pieces of directioninformation relating to a top-and-bottom direction of each of theplurality of strokes, wherein the determining includes determining atop-and-bottom direction of each of the plurality of strokes, by usingat least either a direction of inclination of the device which isdetermined in accordance with an output of an acceleration sensor in thedevice, or a positional relationship between the device and a user whichis determined in accordance with a result of face recognition using animage captured by a camera module of the device; wherein the displayprocessor is configured to select stroke data having a top-and-bottomdirection agreeing with a present direction of inclination of thedevice, from between at least one first stroke data having atop-and-bottom direction which is a first direction, and at least onesecond stroke data having a top-and-bottom direction which is a seconddirection that is different from the first direction, and to display atleast one stroke corresponding to the selected stroke data on thescreen.
 10. The method of claim 9, wherein the positional relationshipbetween the device and the user, which is determined in accordance withthe result of the face recognition, is configured to determine atop-and-bottom direction of each of the plurality of strokes, when theinclination of the device is less than a first inclination.
 11. Themethod of claim 9, wherein the direction of inclination of the device isconfigured to determine a top-and-bottom direction of each of theplurality of strokes, when the inclination of the device is the firstinclination or more.
 12. The method of claim 9, further comprising:converting, by coordinate conversion, a plurality of first strokeshaving a top-and-bottom direction which is a second direction that isdifferent from a first direction agreeing with a direction of thescreen, to a plurality of second strokes having a top-and-bottomdirection which is the first direction; and outputting a character codecorresponding to the plurality of second strokes.
 13. The method ofclaim 9, further comprising: converting, by coordinate conversion, aplurality of first strokes having a top-and-bottom direction which is asecond direction that is different from a first direction agreeing witha direction of the screen, to a plurality of second strokes having atop-and-bottom direction which is the first direction; and searching fora stroke similar to a third stroke which is a search key from theplurality of first strokes, by matching between the third stroke whichis the search key and the plurality of second strokes.
 14. Acomputer-readable, non-transitory storage medium having stored thereon acomputer program which is executable by a computer, the computercomprising a display processor, and the computer program controlling thecomputer to execute functions of: displaying on a screen of the computera plurality of strokes input by handwriting, determining atop-and-bottom direction for each of the plurality of strokes; andstoring in a storage medium handwritten data, the handwritten datacomprising a plurality of stroke data corresponding to the plurality ofstrokes, and a plurality of pieces of direction information relating toa top-and-bottom direction of each of the plurality of strokes, whereinthe determining determines a top-and-bottom direction of each of theplurality of strokes, by using at least either a direction ofinclination of the computer which is determined in accordance with anoutput of an acceleration sensor in the computer, or a positionalrelationship between the computer and a user which is determined inaccordance with a result of face recognition using an image captured bya camera module of the computer; wherein the display processor isconfigured to select stroke data having a top-and-bottom directionagreeing with a present direction of inclination of the device, frombetween at least one first stroke data having a top-and-bottom directionwhich is a first direction, and at least one second stroke data having atop-and-bottom direction which is a second direction that is differentfrom the first direction, and to display at least one strokecorresponding to the selected stroke data on the screen.
 15. The storagemedium of claim 14, wherein the positional relationship between thecomputer and the user, which is determined in accordance with the resultof the face recognition, is configured to determine a top-and-bottomdirection of each of the plurality of strokes, when the inclination ofthe computer is less than a first inclination.
 16. The storage medium ofclaim 14, wherein the direction of inclination of the computer isconfigured to determine a top-and-bottom direction of each of theplurality of strokes, when the inclination of the computer is the firstinclination or more.
 17. The storage medium of claim 14, wherein thecomputer program further controls the computer to execute functions of:converting, by coordinate conversion, a plurality of first strokeshaving a top-and-bottom direction which is a second direction that isdifferent from a first direction agreeing with a direction of thescreen, to a plurality of second strokes having a top-and-bottomdirection which is the first direction; and outputting a character codecorresponding to the plurality of second strokes.
 18. The storage mediumof claim 14, wherein the computer program further controls the computerto execute functions of: converting, by coordinate conversion, aplurality of first strokes having a top-and-bottom direction which is asecond direction that is different from a first direction agreeing witha direction of the screen, to a plurality of second strokes having atop-and-bottom direction which is the first direction; and searching fora stroke similar to a third stroke which is a search key from theplurality of first strokes, by matching between the third stroke whichis the search key and the plurality of second strokes.